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Virtual  Reality,  as  its  name  suggests,  is  an  unreal,  alternate  reality 
in  which  anything  could  happen.  In  its  1991  technological  implementation, 
Virtual  Reality  is  a  3D  video  game  you  can  enter  by  strapping  something  onto 
your  face  that  fools  your  senses  into  perceiving  an  environment  that  surrounds 
you  on  all  sides.  The  thing  strapped  to  your  face  is  called  a  Head-Mounted 
Display . 

The  true  potential  of  this  new  field  comes  from  the  ability  of  a 
Head-Mounted  Display  to  induce  a  synthetic  experience  in  its  wearer.  The 
ability  to  artificially  create  and  design  experience  enables  possibilities  and 
powers  that  were  formerly  impossible.  If  experience  can  be  captured  and 
transmitted,  then  you  can  "travel"  instantaneously  to  a  distant  location  and 
see  the  trees,  feel  the  wind,  hear  the  birds,  and  smell  the  flowers.  If 
electronic  instruments  can  sense  things  that  you  cannot  perceive,  such  as  the 
inside  of  opaque  objects,  then  you  can  be  shown  images  of  these  invisible 
things.  If  microscopes  and  tiny  probes  can  scan  and  manipulate  the  microscopic 
world,  then  you  can  "shrink,"  like  Alice  in  Wonderland,  to  enter  into  a 
three-dimensional  world  of  palpable  bacteria  and  Brontosaurian  insects. 

The  true  potential  of  the  Head-Mounted  Display,  the  piece  of  gear  that 
has  enabled  Virtual  Reality,  is  not  that  it  allows  you  to  enter  into  a  fantasy 
world,  but  that  it  allows  you  new  ways  of  perceiving  the  real  world. 

Expansion  of  Perception 

Vision,  hearing,  touch,  taste,  and  smell  are  the  traditional  five 
senses,  and  in  addition,  you  have  the  ability  to  sense  temperature,  vibration, 
acceleration  of  your  body,  the  positions  of  your  limbs,  forces  acting  on  your 
body,  hunger,  thirst,  pain,  and  other  sensations  related  to  your  body's 

internal  state. 


There  are,  however,  things  which  are  invisible  to  all  of  your  senses. 
Examples  of  these  imperceptible  phenomena  are  X-rays,  infrared  radiation,  radio 
waves,  magnetic  fields,  radioactivity,  ultrasound,  electricity,  the  inside  of 
opaque  objects,  microscopic  objects,  and  events  occurring  too  fast  to  see.  Even 
though  you  cannot  directly  perceive  these  things,  you  can  indirectly  measure 
and  observe  them  with  various  instruments  and  electronic  sensors. 


By  linking  electronic  sensors  to  a  Head-Mounted  Display,  it  is  now 
possible  to  create  "sensory  transducers,"  which  will  allow  you  direct 
perception  of  these  phenomena  which  are  imperceptible  without  electronic 
augmentation.  As  an  example,  night-vision  goggles  allow  you  to  see  and  move 
about  in  total  darkness  by  amplifying  the  low  levels  of  light  that  are  actually 
present.  Sensory  transducers  can  be  built  to  make  visible  radioactivity,  or  any 
other  invisible  phenomenon  for  which  electronic  sensors  exist. 
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Ultrasound  scanners  are  currently  used  to  look  into  the  human  body, 
and  by  connecting  these  scanners  to  a  Head-Mounted  Display,  it  will  soon  be 
possible  to  see  directly  into  the  living  tissue.  By  using  half-silvered 
mirrors,  the  Head-Mounted  Display  can  allow  you  to  see  through  to  the  real 
world,  with  the  image  from  the  ultrasound  scanner  optically  superimposed.  Thus, 
an  obstetrician  examining  a  pregnant  woman  could  see  the  woman,  feel  the  fetus 
kick  beneath  his  hands,  and  see  the  ultrasound  image  of  the  fetus  appearing  to 
hang  in  space  inside  of  her  belly.  We  call  this  "X-ray  vision,"  by  analogy  with 
Superman's  ability  to  see  inside  of  solid  objects,  even  though  it  is  ultrasound 
rather  than  X-rays  than  would  actually  be  used.  Figure  1  shows  a  drawing  of 
this  set-up. 

We  are  working  towards  building  a  prototype  of  such  a  device  here  at 
the  University  of  North  Carolina.  My  colleagues  Henry  Fuchs,  John  Poulton,  John 
Eyles  and  their  team  have  over  the  last  ten  years  built  high-performance 
graphics  computers  (Pixel-Planes)  that  make  it  possible  to  compute  views  of 
two-  and  three-dimensional  medical  image  data  in  real-time.  My  colleague  Steve 
Pizer  and  his  team  have  for  fifteen  years  been  developing  algorithms  and 
systems  to  make  computer  graphics  an  effective  tool  for  radiologists  and 
radiation  oncologists.  The  volume  rendering  technique  they  have  developed  will 
be  useful  for  displaying  3D  ultrasound  image  data.  My  colleagues  Jannick 
Rolland  and  Rich  Holloway  are  designing  the  optics  and  software  for  a 
see-through  Head-Mounted  Display  for  medical  use.  We  hope  to  have  this 
prototype  of  the  X-ray  vision  goggles  working  in  the  next  year  or  so. 

A  Head-Mounted  Display  must  be  head-mounted  because  your  eyes,  ears, 
nose  and  tongue  are  head-mounted.  Your  senses  are  directional,  and  the  location 
in  space  from  which  sensory  inputs  originate  is  an  important  part  of  your 
perception  of  the  world  around  you.  Seeing  or  hearing  a  nearby  rattlesnake  with 
your  natural  senses  implies  knowing  where  it  is  relative  to  your  body.  A  Geiger 
counter  that  warns  you  when  radiation  is  present  is  better  than  nothing,  but  a 
head-mounted  sensor  that  lets  you  see  radioactive  gas  leaking  through  the  wall 
is  a  vast  improvement  in  awareness. 

What  do  these  imperceptible  things  look  like?  Well,  they  don't  look 
like  anything  —  they're  invisible.  A  visual  representation  must  be  invented. 
This  is  a  graphic  design  problem.  What  does  radiation  look  like?  Perhaps  it  is 
purple,  with  a  brightness  that  indicates  its  lethality. 

It  is  useful  to  consider  earlier  efforts  to  portray  the  invisible. 

What  does  a  molecule  look  like?  Since  atoms  are  a  thousand  times  smaller  than 
the  wavelength  of  light,  molecules  have  no  visual  appearance,  and  so  an 
appropriate  graphical  representation  must  be  created.  (The  appearance  of  a 
molecule  under  an  electron  microscope  is  not  a  definitive  answer  to  what  a 
molecule  looks  like,  but  rather  a  technologically  convenient  choice  of  a  visual 
representation.)  Figure  2  shows  two  different  depictions  of  the  same  protein 
molecule  —  one  representation  emphasizes  how  the  molecule  fills  space,  while 
the  other  shows  only  the  path  of  the  proteinUs  polypeptide  backbone  as  it  winds 
through  space. 


Figure  2.  Caption  What  does  a  molecule  look  like? 

Two  representations  of  dihydrofolate  reductase. 

(a)  touching  colored  spheres 

(b)  ribbon  backbone 

My  colleague  Fred  Brooks  has  for  more  than  fifteen  years  here  at  the 
University  of  North  Carolina  been  putting  computer  graphics  to  work  to  help 
biochemists  understand  the  structure  and  properties  of  the  large  organic 
molecules  they  study.  These  representations  are  ones  that  have  proved 
informative  to  the  biochemists. 


What  does  human  speech  look  like?  Casting  speech  into  visible  form  is 
something  you  take  for  granted:  it  is  called  writing.  Various  innovations 
occurred  over  the  last  five  thousand  years  —  the  invention  of  the  alphabet, 
spaces  between  words,  the  printing  press,  and  standardized  spelling.  Writing  is 
now  a  mature  art,  and  the  link  between  the  sounds  of  speech  and  the  black  marks 
on  a  page  of  a  book  are  quite  abstract.  Figure  3  contrasts  (a)  a  sonogram,  a 
straightforward  way  to  visualize  vocal  sound,  with  (b)  the  much  more  abstract 
form  of  modern  written  language. 

Figure  3  Caption:  What  does  speech  look  like? 

(a)  sonogram 

(b)  English  prose 

Reading  is,  in  a  sense,  hearing  with  your  eyes.  This  cross-sensory 
substitution  is  closely  related  to  a  sensory  transducer  for  the  imperceptible. 
For  the  deaf,  the  sound  of  human  speech  is  an  imperceptible  phenomenon.  If  it 
were  possible  to  make  a  device  that,  in  real-time,  converted  speech  to  written 
text,  then  this  device  would,  in  effect,  allow  a  deaf  person  to  hear.  This 
capability  --  real-time,  speaker-independent,  continuous  speech  recognition  -- 
has  not  yet  been  achieved.  It  ought  to  be  possible,  nevertheless,  to  create 
some  kind  of  real-time  visual  representation  of  the  sound  of  speech  that,  when 
visually  superimposed  on  the  movements  of  the  lips  and  face,  is  sufficient  to 
allow  a  deaf  person  to  comprehend  what  is  being  said. 

What  does  computation  look  like?  The  step-by-step  actions  of  a 
computer  as  it  manipulates  data  under  the  control  of  a  program  is  a  dynamic 
process  that  could  be  given  an  animated  graphical  representation,  but  there  is 
currently  no  widely-used  or  accepted  depiction  of  the  process  of  computation. 
Computation  is  currently  invisible. 

The  electronic  expansion  of  human  perception  has,  as  its  manifest 
destiny,  to  cover  the  entire  human  sensorium.  Ultraviolet  rays  that  will  cause 
a  sunburn  hours  later  might  be  mapped  to  an  insistent  vibration  on  the  skin. 
Dangerous  radiation  which  would  kill  you  in  a  few  minutes  might  be  signaled 
directly  with  purposely-induced  pain.  If  each  computer  instruction  were  mapped 
to  its  own  audible  frequency,  then  each  computer  program  would,  because  of  its 
characteristic  sequence  of  instructions,  make  its  own  recognizable  sound.  And, 
as  Julius  Smith  of  Stanford  has  suggested,  if  design  rule  violations  were 
mapped  to  smells,  an  engineer  could  turn  to  his  colleague's  circuit  diagram  and 
truthfully  say,  "This  design  stinks." 

Real-Space  Databases 

Information  is  often  associated  with  location.  Maps,  inventories,  and 
mailing  lists  are,  in  essence,  lists  of  information  about  objects  and  features 
at  specific  locations. 

Using  a  see-through  Head-Mounted  Display  which  tracks  its  location  in 
the  world,  graphical  data  files  could  be  spatially  registered  with  the  real 
world.  A  particular  graphic  object  from  the  data  file  would  be  seen  sitting  at 
one  spot  in  the  world,  and  nowhere  else.  The  data  file  would  give  the 
coordinates  of  the  object's  location  —  a  very  accurate  latitude,  longitude, 
and  height  above  sea  level  —  and  only  at  that  location  could  it  be  seen. 
Michael  Naimark,  a  media  technologist  in  San  Francisco,  has  coined  the  term 
"real-space  imaging"  to  describe  graphics  that  are  registered  with  the  real 
world,  just  ac  real-time  graphics  are  synchronized  with  events  in  the  real 
world . 


To  find  a  specific  item  whose  location  in  a  company's  huge  warehouse 
is  known,  a  huge  red  blinking  arrow  could  appear  in  the  air  above  the  item, 
always  remaining  above  it  as  you  approached.  Ghostly  computer  graphic  labels 
could  be  attached  to  real-world  objects  and  places.  At  specific  places,  you 
could  leave  notes  to  yourself  that  only  you  could  see.  ("Don't  eat  at  this 
place  again.")  You  could  leave  warnings  for  others  or  scrawl  rude  graffiti. 

The  difference  between  these  virtual  labels  and  real  physical  ones  is 
that  everyone  can  see  physical  labels,  whereas  each  virtual  label  exists  in 

some  spatial  data  file  and  can  only  be  seen  if  you  have  loaded  that  file  into 

your  Head-Mounted  Display.  This  means,  unfortunately,  that  virtual  billboards 
will  probably  not  replace  the  physical  ones  that  line  the  highways  —  they 
would  be  too  easy  to  turn  off.  To  guide  you  to  her  house,  a  friend  might  give 

you,  not  written  directions  or  a  map,  but  a  spatial  data  file  that  had  a  huge 

orange  stripe  hovering  ten  feet  above  the  road  along  the  route  from  your  house 
to  hers.  It  would  be  difficult  to  miss  a  turn  at  an  intersection  where  the  huge 
orange  stripe  above  you  veered  to  the  right. 

Geographical  information  systems,  which  have  become  popular  lately, 
are  in  essence  computerized  maps.  Information  is  displayed  in  a  spatial 
distribution  that  conforms  with  the  spatial  distribution  of  the  real-world 
objects  the  information  describes.  If  such  a  computerized  map  is  enlarged  and 
superimposed  onto  the  real  world,  then  it  becomes  a  real-space  database. 

A  real-space  graphical  model  of  a  building  could  be  created  on-site, 
before  construction  began,  to  see  what  the  building  would  look  like  in  its  real 
surroundings.  Figure  4  shows  a  drawing  of  this.  An  architect  and  client  could 
walk  through  a  simulated  house  at  its  planned  site,  looking  through  simulated 
windows  at  the  real  trees  surrounding  the  site.  The  architectural  walk-through 
team  here  at  the  University  of  North  Carolina,  led  by  Fred  Brooks,  has  for 
several  years  been  building  detailed  computer  models  of  buildings  and  exploring 
the  questions  of  how  realistic  the  models  need  to  appear,  how  the  user  can  move 
through  them,  and  how  they  can  be  segmented. 

The  plans  and  maintenance  instructions  for  a  complex  mechanism  such  as 
an  aircraft's  jet  engine  could  be  spatially  superimposed  on  the  engine  being 
repaired,  with  the  engine's  self-diagnostic  circuits  causing  a  large  red  arrow 
to  point  to  the  particular  part  that  required  replacement. 

In  time,  a  huge  number  of  real-space  databases  will  come  to  exist,  and 
just  as  you  must  choose  which  books  you  will  read,  you  will  have  to  choose 
which  graphic  databases,  if  any,  you  wish  to  overlay  onto  the  world. 

Remote  Presence 

Being  somewhere,  from  an  experiential  point  of  view,  consists  of  beina 
able  to  look  around  at  the  things  that  surround  you,  to  touch  them,  to  walk 
around,  and  to  hear,  feel,  and  smell  whatever  is  present  with  you.  If  the 
light,  sound,  and  other  physical  phenomena  which  trigger  your  senses  can  be 
detected  electronically  and  transmitted  to  a  Head-Mounted  Display,  then  it  is 
possible  to  have  the  experience  of  being  at  a  place  when  in  fact  your  body  is 
many  miles  away.  This  technological  out-of-body  experience  is  called 
"telepresence . " 

The  experience  of  simulated  presence  at  a  distant  place  is,  in  fact, 
very  familiar  to  each  of  us.  When  using  a  telephone,  your  voice  and  ears  are 
electronically  linked  to  those  of  a  distant  person,  and  you  converse  as  if  you 
were  in  the  same  room.  This  is  auditory  telepresence.  It  seems  very  natural  and 
norma  1 . 


To  have  the  visual  experience  of  a  3D  world  that  surrounds  you,  it  is 
necessary  to  see  with  both  your  eyes  and  to  be  able  to  look  around.  The 
technical  implementation  of  visual  telepresence  is  to  feed  the  signals  from  a 
pair  of  video  cameras  to  the  left-  and  right-eye  fields  of  a  Head-Mounted 
Display,  with  the  cameras  mounted  in  a  robot  head  whose  motion  mimics  that  cf 
your  own  head.  When  you  turn  your  head,  the  robot  head  turns.  And  since  you  see 
what  the  robot  head  sees,  when  you  look  around,  you  see  the  environment  that 
surrounds  the  robot.  The  "Green  Man"  is  a  robot  built  for  underwater  remote 
presence  research  by  the  Naval  Ocean  Systems  Center  in  Hawaii.  (It  is  called 
the  Green  Man  because  its  movements  are  hydraulically  actuated,  and  the  green 
hydraulic  fluid  drips  from  its  hoses  as  if  it  were  green  alien  blood.) 

In  addition  to  cameras  slaved  to  the  motion  of  your  head,  the  Green 
Man  also  has  robot  arms  slaved  to  the  motions  of  your  own  arms.  By  putting  on 
the  gloves  and  headset  that  link  you  to  the  distant  robot,  your  senses  are 
transported  into  the  robot  body.  If  your  eyes  and  hands  are  at  a  remote 
location,  you're  there. 

When  you  control  such  a  telepresence  robot  from  a  distance,  you  can 
see,  hear,  and  converse  with  another  person  in  front  of  the  robot.  The  person 
you  talk  to,  however,  would  not  see  you,  but  rather  a  slimy  green  robot  that 
was  gesticulating  and  talking  with  your  voice. 

In  a  telephone  conversation,  each  person  has  a  microphone  and  a 
speaker  in  the  telephone  handset,  with  each  microphone  linked  to  the  other 
person's  speaker.  The  analogous  setup  for  visual  telepresence  is  for  each 
person  to  wear  a  see-through  Head-Mounted  Display  which  is  linked  to  a 
telepresence  robot  at  the  location  of  the  other  person.  In  such  a  conversation, 
you  would  see  the  other  person' s  face  superimposed  onto  the  robot  face  which 
was  physically  there  before  you.  You  could  thus  have  a  "face-to-face" 
conversation  with  a  distant  person,  achieving  eye  contact  and  observing  one 
another's  facial  expressions. 

The  telephone  allows  electronic  ventriloquism.  It  lets  you  throw  your 
voice,  at  the  speed  of  light,  to  any  location  where  you  can  ccax  someone  to 
pick  up  the  receiver.  Likewise,  visual  telepresence  will  allow  you  to  project 
your  eyes,  at  the  speed  of  light,  to  any  location  where  a  telepresence  robot 
exists.  This  is  instant  travel.  An  executive  might  in  ten  minutes  do  his  daily 
tour  of  the  warehouse,  the  factory,  the  lab  and  the  accounting  department,  even 
though  these  places  are  thousands  of  miles  apart. 

The  robot  hands  of  the  telepresence  robot  a'  .ow  the  human  operator  to 
manipulate  objects  at  the  robot's  location.  This  g,es  beyond  the  mere  passive 
sensory  "presence"  at  a  remote  location  and  therefore  has  a  different  name  -- 
"tele-operation."  In  extremely  dangerous  environments,  mortal  human  beings  car. 
be  replaced  with  human-controlled  tele-robots.  The  Green  Man  was  designed  to 
work  on  the  bottom  of  the  ocean,  too  deep  for  divers.  NASA  may  use 
tele-operated  robots  to  construct  its  space  station.  Tele-manipulators  are  used 
to  handle  the  radioactive  fuel  in  nuclear  reactors.  Tele-operated  robots  are 
beginning  to  be  used  to  fight  fires  and  to  defuse  bombs. 

Micro-Tele-Operation 


Tele-robots  don't  have  to  be  the  same  size  as  their  human  operators. 
Tele-robots  the  size  of  King  Kong  could  be  made,  say,  for  constructing 
buildings.  Tiny  tele-robots  coi  Id  also  be  made.  As  operator  of  a 
miur u-ceie-robot ,  you  would  have  the  perception  that  the  ordinary  world  had 
expanded  enormously,  or  equivalently,  that  you  had  been  miniaturized.  Operating 
at  a  l-to-100  scale  factor,  the  micro-robot  would  be  two  centimeters  high  and 
you  would  perceive  a  mouse  to  be  the  size  of  an  elephant. 


Some  work  has  already  been  done  in  micro-tele-operation.  A 
scanning-tunneling  microscope  can  image  individual  atoms,  detect  surface  forces 
as  it  probes  these  atoms,  and  move  atoms  around  with  its  probe;  at  an  IBM  lab 
in  Yorktown  Heights,  New  York,  researchers  have  hooked  up  such  a  microscope  to 
a  force-feedback  device  to  ,.ake  it  possible  to  "touch"  atoms.  Controlling 
micro-robots  is  one  of  the  goals  of  Dr.  Tachi  in  Tsukuba  Science  City,  Japan, 
who  is  one  of  the  leading  researchers  in  one-to-one  scale  tele-operation. 

It  is  not  necessary  to  make  smaller  and  smaller  self-contained  robots 
to  achieve  micro-tele-operation.  With  a  microscope  and  micro-manipulator,  you 
could  effectively  have  your  eyes  and  hands  projected  into  the  microscopic 
world.  And  to  perceive  that  the  micro-world  surrounded  you,  when  you  turned 
your  head,  the  microscope  would  need  to  swivel  around  the  specimen  to  achieve 
the  right  point  of  view.  Another  way  to  achieve  quick  changes  of  point  of  view 
in  the  micro-world  would  be  to  mount  the  specimen  on  an  electrically-controlled 
rotation  stage  beneath  the  lens  of  a  fixed  microscope.  This  approach  assumes  a 
micro-world  that  is  relatively  transparent,  such  as  a  drop  of  water  from  a 
pond,  so  that  any  internal  point  of  view  can  be  achieved,  even  though  the 
microscope  looks  in  from  the  outside. 

Scaled  down  by  a  factor  of  100,  you  could  reach  out  and  tweak  the 
antenna  of  a  honeybee  that  you  perceive  to  be  four  feet  long,  as  shown  in 
Figure  6.  And  it  couldn't  sting  you. 

An  effective  micro-tele-robot  could  be  used  for  micro-surgery.  An 
adventurer  could  take  a  microscopic  safari  into  an  anthill  to  battle  the 
furious  hordes  of  ants.  As  people  begin  to  work  and  play  -in  micro-worlds,  a 
need  will  arise  for  microscopic  tools  and  devices  which  will  perhaps  be 
manufactured  using  micro-tele-robots. 

Virtual  Reality  will  prove  to  be  a  more  compelling  fantasy  world  than 
Nintendo,  but  even  so,  the  real  power  of  the  Head-Mounted  Display  is  that  it 
can  help  you  perceive  the  real  world  in  ways  that  were  previously  impossible. 

To  see  the  invisible,  to  travel  at  the  speed  of  light,  to  shrink  yourself  into 
microscopic  worlds,  to  re-live  the  experiences  of  yourself  and  others  —  these 
are  the  powers  that  the  Head-Mounted  Display  offers  you.  Though  it  sounds  like 
science  fiction  today,  tomorrow  it  will  seem  to  you  as  commonplace  as  talking 
on  the  telephone. 
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